Modeling Conversational Speech for Speech Recognition

In language modeling for speech recognition the goal is to constrain the search of the speech recognizer by providing a model which can, given a context, indicate what the next most likely word will be. In this paper, we explore how the addition of information to the text, in particular part of speech and dysfluency annotations, can be used to,build more complex language models. In particular, we ask two questions. First, in conversational speech, where there is a less clear notion of "sentence" than in written text, does segmenting the text into linguistically or semantically based units contribute to a better language model than merely segmenting based on broad acoustic information, such as pauses. Second, is the sentence itself a good unit to be modeling, or should we look at smaller units, for example, dividing a sentence into a "given" and "new" portion and segmenting out acknowledgments and replies. To answer these questions, we present a variety of kinds of analysis, from vocabulary distributions to perplexities on language models. The next step will be modeling conversations and incorporating those models into a speech recognizer.